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ABSTRACT 

Evaluating alternatives provides both the reason for 
and the technique of program budgeting for educational planning. The 
activities demanded by the program budgeting system allow systematic 
choosing of a preferred course of action. Within this system, 
alternatives are considered in the context of all other programs. 

This document presents concepts, techniques, and problems involved in 
educational program analysis. Data from a developmental program in 
the San Jose Unified School District are presented as a basis for the 
discussion. (Author/RA) 
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ANALYSIS OF EDUCATIONAL PROGRAMS WITHIN 



A PPOGRAM BUDGETING SYSTEM 

* 

M. B, Carpenter 
S. A, Haggart 

The Rand Corporation, Santa Monica, California 



INTRODUCTION 



Evaluating alternatives is both the why and the how of program 
budgeting for educational planning. Being able to systematically 
choose the preferred course of action is the purpose of all the activ- 
ities demanded by the program budgeting system. Within the system, 
alternatives are considered in the context of all other programs, not 
in isolation. For example, the cost of the alternative is calculated 
as an incremental cost to the cost of all other district programs. 

It is also conceivable that an alternative program could result in a 
less costly program — a rare occurrence in recent years. 

We will discuss the concepts, techniques, and problem areas in- 
volved in the how of analyzing educational programs. As a basis for 
discussion, we will use the data from a developmental program being 
carried out in the San Jose Unified School District under Senate Bill 
28. The example will illustrate most of the aspects of analysis. 



Any views expressed in this paper are those of the authors. 

They should not be interpreted as reflecting the views of The Rand 
Corporation or the official opinion or policy of any of its govern- 
mental or private research sponsors. Papers are reproduced by The 
Rand Corporation as a courtesy to members of its staff. 

This material was presented on August 14, 1969, during the morn- 
ing session, which was part of a 5-day program, ’’Program Budgeting 
(PPBS), A Resource Allocation Decision System for Education,” given 
as an Academy Clinic by the American Association of School Adminis- 
trators/National Academy for School Executives. 
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THE FRAMEWORK FOR A]:^ALYSIS 



An alternfitive prc5ram may be described In terms of certain basic 
characteristics — its effectiveness, its cost, and its resource require- 
ments. The goal of analysis is not to provide the decisionmaker with 
the alternative that maximizes or minimizes specific characteristics; 
the goal is to provide information which together with the judgment of 
the decisionmaker permits a compromise among the characteristics of 
the alternative within the various environmental constraints, such as 
budget level. 

An understanding of the framework of the analysis and the ground 
rules of the analysis is as important as the analysis itself. For 
this reason, we will describe very briefly the context of analysis. 

This may best be achieved by looking at the\ elements of analysis, the 

I 

process of analysis, and the nature of a **b<itter" analysis. We will 
also discuss the specific technique of cost/effectiveness analysis. 

There are six major elements of the analysis: 

Objectives* What educational aims are we trying to accomplish 
with the resources that the analysis is designed to compare? The 
choice of objectives is fundamental; if the wrong objective is chosen 
the whole analysis may be addressed to the wrong question, 

Altevnatives * By what alternative combinations of resources may 
the objective be accomplished? The generation of new and sometimes 
better alternatives is often an important by-product of the analytical 
part of the program budgeting process; this is partly due to the inter- 
action of the decisionmaker and the analyst, but mainly due to the 
fact that additional analysis often results in a better understanding 
of the problems and the subsequent discovery of other altev natives. 

Costs or Besources Used* Each alternative method of accomplish- 
ing the objective involves incurring certain costs or using certain 
resources. The resources (and cost) required for each alternative 
must be determined. 

A Model or Models* Models are abstract representations of sig- 
nificant relations in the real world that may be manipulated and used 
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to predict others. The model is useful in tracing the relation be- 
tween inputs and outputs, resources and effectiveness, for each alter- 
native to be compared. ' 

Criteria. By criteria we mean the rules by which we choose one 
alternative rather than another. They arc the standards against which 
we measure the contribution of a program toward meeting the program 
objective. 

Effectiveness Measures. These measure the contribution of the 
alternative toward meeting the objective. Ideally, they are not mea- 
sures of workload, although in some cases this may be necessary. 

The elements of the analysis then become inputs to the analytical 
process (Fig. 1). The process begins, of course, with the alternatives 
to be evaluated. These are examined within the model that represents 
the input-output or the resources-ef f ectiveness relationships of the 
system. It tells what can be expected from each alternative. Essen- 
tially, it shows the cost of the alternative and the contribution of 
the alternative in meeting an objective. A criterion is then used to 
weigh the cost against performance* The purpose is not to determine 
one ratio of effectiveness to cost for an alternative but rather to 
rank alternatives. This information then provides a part of the basis 
for selection among alternatives.. This is the quantified information 
for decisionmaking. 

There is another important aspect of this process: it is the 

consideration of those dimensions of the problem that cannot be quan- 
tified — the qualitative or intangible factors vrhich make it impossible 
to define a single satisfactory criterion. 

It is not possible to say what makes a good analysis; it is pos- 
* sible to say what might help to make a better analysis. The fact that 
a set of procedures or exact rules still remains elusive is not a good 
reason to give up the search for ways to make the best of what is 
available. The search continues — with the dialogue between opponents 
and proponents helping most to clarify both the problems and the solu- 
tions. Several actions may help in the quest for a better analysis. 
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Adapted from E. S. Quade, Systems Analysis Techniques for Planning -Frogramming-Budgeting , 

The Rand Corporation, P-3322, March 1966. 
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First, be sure that the right problem being addressed, that 
the analysis of the problem is seeking the answer to the right ques- 
tiont This includes not only deciding alternative ways to do something 
but also deciding what should be donee The right answer to the wrong 
question is, after all, rather useless • Important in deciding on the 
right problem is considering the interactions of other activities as 
they have a bearing on the problem. The complicating fact is that the 
really difficult problems are not isolated from their environment, and 
their solutions cannot be sought in isolation. 

Assume success in divining the right problem to be studied and 
in designing the analysis to seek the answer to the right question. 

What is left? There is one important area over and above the analysis 
per se. It is the presentation of the results. 

In the presentation of the analysis, the qualitative considera- 
tions should be identified. This includes both those taken into ac- 
count in the analysis and those that could not be made an integral 
part of the analysis itself. It is important to present the results 
of the formal quantitative analysis, and interpret the results with 
special attention to the assumptions and limitations of the analysis. 

In addition, the analyst should attempt to identify the important qual- 
itative considerations that the decisionmaker should try to take into 
account . 

One additional thought should be mentioned. Analysis does not 
necessarily mean number juggling. A great deal can be gained from 
just a systematic approach to defining the problem and seeking possi- 
ble solutions. Numbers, of course, do help. We all know that. We 
also know that some numbers are better than other numbers. The trick 
is to know as well as possible the meaning of the numbers: What do 

they tell you? Where do they come from? On what are they based? The 
point that should be emphasized is that numbers alone do not make a 
better analysis; the important facet is the* context in which they are 
used and how they are used. The process of trying to make explicit 
some of the qualitative considerations inherent in defining the prob- 
lem and in seeking possible solutions probably contributes more to mak- 
ing a better analysis. 
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The cost-effectiveness technique of comparing alternatives is the 
most widely known and the most often misused technique. It is useful 
in comparing alternatives when either the cost (budget level) or the 
effectiveness (achievement) is held constant. Maximizing the ratio 
for the sake of the ratio alone can lead to some ridiculous extremes — 
like zero to infinite cost or zero to infinite effectiveness. But 
there is a way to make the technique meaningful. That is, for example, 
to specify the level of effectiveness and then examine the cost of 
alternative means to achieve that effectiveness. Conversely, you can 
fix a single budget Level and examine the levels of effectiveness that 
can be achieved through different alternatives. The ratio itself can 
be a very simple guide to ranking alternatives when two conditions 
exist: (1) the scale of the activity is fixed, and (2) the alterna- 

tives are not interdependent. 

There are four purposes for which cost-effectiveness analysis can 
be used. The first is in the allocation of resources among major ob- 
jectives. This is really a variation of the second and third purposes- 
the choice of alternative means to meet any given, objective, and the 
assessment of the merit of different objectives. The fourth purpose 
is to provide the systematic generation of alternatives not initially 
identified. 

We have chosen, in the following illustration, to examine the 
alternatives on an equal-cost basis. This approach focuses the atten- 
tion of both the decisionmaker and the analyst on the effectiveness of 
the alternatives — the more difficult area in the cost-effectiveness 
analysis activity. 

The next section provides a description of the developmental pro- 
gram, the R-3 Project, used as the illustration. This is followed by 
a discussion of the generation of equal-cost alternatives and a dis- 
cussion of comparing the effectiveness of equal-cost alternatives. 
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DESCRIPTION OF PROJECT R-3 

There are 35,000 students in the San Jose Unified School District, 
in which Project R-3 is being conducted, San Jose, California, is in 
the center of an area characterized by a large Mexican-Araerican popula- 
tion and by rapid industrialization. In some neighborhoods in downtown 
San Jose, the concentration of Mexican-American residents can run as 
high as 70 percent. But prejudice against Mexican-Americans is slight 
compared to what one might expect to find in, say, Texas, so that there 
are job opportunities for ambitious Mexican-Americans in the San Jose 
area. 

Population growth in San Jose has been more rapid than in the 
state as a whole, which, as you know, is itself characterized by rapid 
population growth. Population growth has had two effects. One is the 
loss of large areas of orchards and farmlands. In the past San Jose 
was known as a center for fruits and vegetables, but now many of the 
orchards are being cut down to make room for new homes* The other 
effect has been a large increase in nonagricultural employment, which 
more than tripled between 1950 and 1966, It is of particular interest 
that the electronics industry had become the largest single employer 
by 1966 in a town in which the canning industry had at one time been 
the largest employer. 

The R-3 Project was funded under California Senate Bill 28, which 
was passed shortly after the Watts riot. Part of that bill was aimed 
at improving the achievement of students in the 7th, 8th, and 9th grades 
who were at least 1 year below grade level in reading or math and who 
were judged to be capable of doing better. Such children are charac- 
terized as underachievers, 

"R-3" stands, not as you might think, for the three Rs — reading, 
writing, and arithmetic — but for readiness, relevance, and reinforce- 
ment, The concept behind the project is that the student is ready to 
learn when he is motivated; motivation is produced by showing the rele- 
vance of learning to the world of work; learning is made more lasting 
by reinforcement of acts which promote cognitive and affective develop- 
ment. 
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The objectives of the program were both short term and long term. 
The short-term objectives were to raise the students* achievement in 
reading and math beyond the normal for the target population, and to 
induce positive attitudes toward learning and education. The long-term 
objectives were to raise the educational and vocational aspirations of 
the students. - * 

Three groups of students were chosen for the experiment. The pro- 
gram group which received the special treatment was chosen at random 
from a group of students in the 8th grade at Woodrow Wilson Junior High. 
These students were no more than 2 years nor less than 1 year below 
grade level in reading or math. (Actually, some of the students were 
above grade level in one or the other of these subjects.) The other 
criterion for student selection was chosen so that the project could 
be carried to successful completion. It was that the students would 
not be likely to move out of the district during the school year. Using 
these criteria, 17 boys and 17 girls were chosen for the program group. 

The control group for comparison of academic achievement was cho- 
sen from another school in a similar socioeconomic area. This school 
had the same tracking program as Woodrow Wilson Junior High. All of 
the students were taken from the Y- track, which comprised students of 
average intelligence who were underachieving. No one in the district 
knew who the students in the control group were. 

The third group was chosen for comparison on the basis of indica- 
tors of attitude toward school — primarily records of attendance and 
disciplinary problems. The comparison group consisted or all Woodrow 
Wilson 8th graders who qualified for the R-3 program but were not in it. 
The group was chosen from Woodrow Wilson students because the project 
personnel knew the backgrounds and environments of the students in this 
group, and also because these students would be treated with the same 
administrative policies for suspensions and expulsions as would the 
students in the program group. 

The project had several components that could grouped in a num- 
ber of ways. We have chosen to break the project into three parts: 
remedial reading and math, the study of occupational technology, and 
the involvement of parents and students in special activities. We shall 
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describe each of theise briefly^ The subject-matter content of the re- 
medial reading and math was not changed from that of the standard 8th 
grade curriculum. Each was given during one of two periods to classes 
of 15 students each, approximately half of the size, of a normal class, 

A diagnostic/prescriptive approach was used. Initial diagnosis of read- 
ing difficulties wrs made by means of the Durr ell reading test, and of 
math from the profiles of the students* performance on subtests of the 
California Achievement Test in math. 

Occupational technology was taught through a variety of means. 

In the classroom, gaming and simulation were used with groups of approx- 
imately 15 students each. This activity was geared to the reading and 
math curricula and took one period every day. The gaming/ simulation 
activity, which is how we will refer to this in the future, was a high].y 
structured representation of real-world situations. Students played 
the roles of actual people, such as a park director or a highway engi- 
neer, Each unit was supplemented by a study trip to a facility directly- 
related to classroom work. There were about 19 study trips throughout 
the year. Students helped to make the arrangements for the study trips 
by use of a conference phone. 

The third component was the involvement of parents and students in 
special activities. We* 11 refer to this as 'involvement from here on. 

For involvement of the students, there were two study trips of 4 days 
each. Last year one trip went to Asilomar, a beautiful park on the 
Monterey peninsula; the other went to Big Sur, a park on the coast fur- 
ther south that contains a grove of coast redwood. These study trips 
were again very highly structured. They were intended to break down 
the stereotyped roles of students and teachers in the classroom and to 
involve students in a prolonged and intensive learning experience. It 
was apparent that they accomplished both of these goals. College stu- 
dents were used as team leaders for the learning activities. 

The parents of the students in R-3 were also involved in the pro- 
gram. Before school opened, they were asked to attend a preschool din- 
ner, where they were told what the program was to be about and where 
their consent was sought for the students to participate in the program 
in general ^L^d in the intensive involvement trips in particular. Since 
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this was a Mexican-American community, project personnel were afraid that 
a number of the parents would not approve of their girls going on the in- 
tensive involvement trips and thereby violating some of the traditions 
surrounding Mexican-American girls. However, this did not come to pass. 
In addition, the parents were invited to all of the study trips that 
accompanied the gaming/ simulation activity, to the intensive involvement 
trips, and to several other dinner meetings throughout the year. At 
every one of these activities the parents participated along with the 
students and teachers. For example, they played some of the games dur- 
ing the dinner meetings, and whereas the students were teaching the 
parents at the beginning, the parents were taking great pride in teach- 
ing their children toward the end of the sessions. Finally, the 
teachers made home visits to all of the parents during the course of 
the year to discuss some activity connected with their children’s par- 
ticipation in the program. This assured that each visit had a clear 
purpose so that the parents were at their ease. 

The results of the project can be expressed in terms of both its 
cost and effectiveness. The cost figures, however, are not completely 
relevant for planning purposes because the project was not originally 
designed with the objective of keeping cost doTO. Therefore, many of 
the costs are considerably higher than they probably need to be- For 
example, teachers in one of the oth^ir schools in the district became 
so interested in the intensive involvement trips that they decided to 
design their own. Many teachers donated their time to planning the 
trip, surplus foods were used, and instead of staying at the dormatories 
at Asilomar where the R-3 program was housed, the students went camping. 
The result was that this trip cost only a third as much as a trip for 
the R-3 program. Of course, because the students were camping — and if 
any of you have ever been camping, you will know this is true — most of 
the time had to be spent in housekeeping chores so that the learning 
experience was not as concentrated as it was on the R-3 trips. 

We shall express the effectiveness of the project in two ways. 

The first, displayed on Fig. 2, is academic achievement. You will note 
that in both reading and math the program and control groups were fairly 
well matched at the beginning, but that by the end of the year the 
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AVERAGE READING GRADE PLACEMENT (FROM CAT) 





Boys 


Girls 




R-3 


C-3^ 


R-3 


C-3 
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6.7 


6.6 


6.9 


6.4 


Post 


8.4 


7.9 


8.9 


7.5 


Gain 

1 


1.7 


1.3 


2.0 


1.1 



AVERAGE ARITHMETIC GRADE PLACEMENT (FROM CAT) 
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6.5 


6.9 


6.7 


Post 


7.9 


7.0 


8.3 


7.5 


Gain 


1.2 


0.5 


1.4 


0.8 



C— 3 denotes the control group. 



Fig. 2 — Academic gains 
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program group had gained more in both reading and math than had the 
control. These gains were significant at the 5 percent level. We want 
to point out that the achievement testing was '^iven under standardized 
conditions and under the supervision of a counselor. 

We have only rough indicators for attitude change. These are 

A 

given in terms of attendance and suspensions (Fig. 3). We expressed 
these as a rate^ that is, number of absences per student-day and num- 
ber of suspensions per student-day, in order to take account of the 
different lengths of time and the different numbers of students in the 
fall and spring semester and in the R-3 and the comparison group. We 
have no way to measure the significance of these differences. We note 
that both the program and comparison group had more absences and sus- 
pensions in the spring semester than they did in the fall, and we also 
note that this change was not as great for the program group as it was 
for the comparison group. 

Other results are of interest. One of these had to do with parent 
attendance at the program functions. This averaged 85 percent for all 
program functions, compared with about 16 percent for PTA meetings and 
other school functions. Of great importance to the school and the com- 
munity was the improvement in the school image in the community. School 
personnel knew that professional agitators were trying to stir up trou- 
ble against the school, but for the past 2 years the agitators have been 
unsuccessful. The school principal attributes their lack of success 
primarily to the influence of the R-3 program. Finally, the school has 
gained a reputation for innovation within the district and among other 
California school districts evidenced by many visits to the school by 
people interested in the program. 

The progrf*m has been given in toto up to this point, and there is 
no way, therefore, to know which of the components we have discussed 
is primarily responsible for the achievement gains and for the attitude 
change. Project personnel would very much like to experiment in order 



Pre- and post-tests of attitudes toward school and career were 
also administered to the program group. Although these tests did indi- 
cate attitude changes in the desired directions, the reliability and 
validity of the measures are not known. 
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C-3 .151 



Fig, 3 — Attitude indicators 
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to isolate the effects of the various components. Then a school dis- 
trict wanting to institute a program of this kind could choose those 
components which were most effective, if any of them are without the 
others. This experiment, however, may not be funded. Nevertheless, 
the project personnel are going to continue to follow the students who 
have been in the program. Information will be gathered on their aca- 
demic progress, including the courses chosen in high school as well as 
their grades and scores on standardized tests; and on student attitude, 
primarily as evidenced by suspensions, expulsions, attendance, dropouts, 
police contacts, and clubs joined. If possible, patterns of college 
entrance and vocational choice will be noted, for of course these are 
what the progri.m primarily was aimed to change. 
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GENERATION OF EQUAL-COST ALTERNATIVES 



In using the R-3 Project as a basis for illustrating the analysis 
of educational programs, we have made every effort to convey the intent 
of the demonstration program. We have, however, taken great liberties 
with the cost of the program. Because of this a word of caution is 
necessary at the very beginning; The cost of the alternatives, of their 
components, or of the specific items of equipment used in this illustra- 
tion does not reflect the cost of the program just completed in the San 
Jose Unified School District. Our purpose is to show how cost analysis 
may be done so that the results are useful in the evaluation of alterna- 
tive programs. The emphasis will be on analysis for planning rather 
than on accounting for budgetary or financial controls 

This distinction is an important one. It demands an approach quite 
different from the traditional cost accounting procedures. The most 
basic difference concerns what is included in the cost estimate of the 
program. This means we are really interested in resource analysis as 
contrasted to cost analysis; we want to know what the dollars are buy- 
ing. Having a cost-per-student measure of alternative programs is not 
sufficient. For decisionmaking purposes we want to know the require- 
ments for special teachers, new equipment, additional facilities, and 
so on. We can then translate these requirements into an estimate of 
cost. 

In analysis for planning, we might use techniques for translating 
requirements into cost estimates that would be unsatisfactory from a 
cost accounting point of view. In developing an estimate we would be 
happy using, say, $12,000 per year for a teacher — without breaking the 
$12,000 down by the amounts for each regular budget appropriation cate- 
gories such as instruction, retirement fund, or fringe benefits. This 
kind of breakdown is, of course, required in cost accounting for budget- 
ary control or accountability. But our main concern is to provide in- 
formation about the resource demands of the alternative programs and 
to use a monetary measure of these demands as a convenient way to com- 
pare alternative configurations of the R-3 Project. 
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The R-3 Project of our illustratioa may be described in terms of 
three basic components: the remedial reading-and-mathematics component; 

the intensive involvement of the students, with parental involvement; 
and the gaming/ simulation component. We have identified various mixes 
of these components as options. These, along with the cost for units 
of 30 students, are shown in Table 1. 

Table 1 

COST OF OPTIONS 
(for units of 30 students) 

Components of the R-3 Project 

#1 Remedial Reading and Mathematics 

#2 Intensive Involvement and Parental Involvement 

#3 Gaming/Simulation 

Options 



A Components #1, #2, and #3 $27,130 

B Components #2 and #3 18,990 

C Component #1 only 13,140 

D Component #2 only 8,675 

E Component #3 only 10,315 

F Components #1 and #3 18,455 



The following assumptions are used in the calculation of the esti- 
mated cost of each option and in the generation of the equal-cost alter- 
natives. For each component requiring classroom instruction, a regular 
class of 30 students is separated into two classes of 15 students each. 
Each classroom is remodeled in a sense. The floors are carpeted, walls 
are painted, and new furnishings are added. This is in addition to the 
special furniture (trapezoidal tables, for example) and equipment of 
each component. The classrooms can be used for six periods per day; 
utilization rates above this level require the preparation and outfit- 
ting of an additional classroom. The estimate for this is $3,000 for 
the classrooms and $2,000 for the furniture. 

Instruction estimates are based on one incremental period for every 
two classes of 15 students. That is, one class of 15 is taught by the 
teacher of the regular 30 students and the other class of 15 is taught 
by the additional teacher. No special training is required for the 

A ^ 
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teachers. And there is no special qualification needed to teach the 
classes of any component. It is assumed that the cost of instruction 
is $11,715 per year, based on a five-period instructional day. This 
figure includes the salary of the teacher and all fringe benefits. 

The estimate for inst^'uction is, on this basis, $2,345 a class period. 
These supporting cost details are given in Table 2. The cost of equip- 
ment and materials is also given in Table 2. 

The cost related to the classroom is separated from the cost re- 
lated to the number of students that use the classroom (perhaps six 
different classes of 15 students each). The cost related to the class- 
room (shown in Table 3) varies from $5,160 for Option E, Gcming/SimCLa-^ 
tion Only j to $8,660 for Option A, All Components^ The cost related 
to the number of students receiving classroom instruction (Table 4) 
varies from $4,705 for Option E to $18,470 for Option A. The cost of 
the involvements (for groups of 30 students) is also shown in Table 4. 

We now have enough information to look for some equal-cost alter- 
natives. In evaluating the effectiveness of the alternatives, we have 
chosen Option A as the basic alternative for comparison. This is the 
option that includes all the components of the R-3 Project: the reme- 

dial reading and mathematics, the intensive involvement of the students, 
the parental involvement, and the gaming/ simulation component. The 
following alternatives are selected: 



Alternative I: 


Option C 


for 


90 


students 


$24,320 


II : 


Option D 


for 


90 


students 


25,025 


III: 


Option E 


for 


150 


1 students 


29,135 


IV: 


Option F 


for 


60 


students 


28,750 



The cost of each of these alternatives is shown in Table 5 and is within 
10 percent of the cost of Option A. This is simply an arbitrary rule 
chosen for this illustration. Notice that it eliminates Option B, at 
a cost of approximately $32,000, from consideration as an equal-cost 
alternative. Option C for 90 students makes use of the classrooms for 
the six periods of the day; additional students would require the re- 
modeling and outfitting of additional classrooms and would make the 
cost of Option C more than 10 percent over the cost of Option A. This 
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Table 2 

SUPPORTING COST DETAILS 



Item 


Unit Cost 
($) 


Cost/30 Students 
($) 


Remodeling Classrooms 


3,000 


— 


Furniture 


2,000 


— 


Classroom Materials 






Reading & mathematics 


350 


— 


Gamlng/slmulatlon 


50 


— 


Classroom Equipment 






Reading & mathematics 


2,200 


— 


Gaming/simulation 


560 


— 


Equipment for Involvement 


500 


— 


Materials for Student Use 






Reading 


~ 16 


475 


Mathematics 


~ 14 


425 


Gamlng/simulatlon 


~ 12 


360 


Intensive Involvements 


— 


6,775 


Parental Involvements 


— 


1,400 


Gaming /Simulation Trips 


— 


2,000 


Instruction (per period) 


2,345 


— 
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This is the cost of equipments for both games of the intensive involvements — the Land Grant 
Game and the Oceanography Game. 
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Based on approximately 20 trips per year with an average cost of $100 per trip. 

Based on experience data with 30 students. This includes the cost of 3 dinner meetings with 
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would result in the elimination of Option C for more than 90 students 
as an equal-cost alternative. 

From what might be called the decision matrix of Table 5, we have 
selected the set of equal-cost alternatives used in the following dis- 
cussion on comparing the effectiveness of alternative programs. We 
could have just as easily selected a different set for a budget level 
of about $45,000. 

In an effort to keep this discussion brief, we have given just the 
flavor of the ideas involved in making the results of cost analysis use- 
ful in the evaluation of alternative courses of action. The important 
consideration is that of providing the decisionmaker with information 
about the resources needed for each alternative as well as the cost of 
the alternative. In an actual evaluation you would need much more de- 
tailed information than we have given in this illustration. You would 
need a good picture of your existing resources, how they are used and 
with what effectiveness. This is necessary not only to determine the 
incremental cost of the specific alternative but also to compare the 
effectiveness of the alternative with the effectiveness of current pro- 
grams. Information of this nature is readily available within an opera* 
tional program budgeting system. In fact, organizing the informational 
data base to permit the systematic evaluation of alternatives is the 
primary objective of a program budgeting system* 
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COMPARING THE EFFECTIVENESS OF EQUAL-COST ALTERNATIVES 

First, we shall discuss the primary objectives of each of the 
components of the R-3 program. Originally, the program was designed 
so that all of the components would support one another in attaining 
the two objectives; improved achievement in reading and math, and an 
improved attitude toward education. But when we consider each compo- 
nent separately, it does seem that we can identify one or the other 
of these two objectives as being the primary objective of a given com- 
ponent, For example, the primary objective of the reading and math 
program was clearly to improve the students’ academic achievement in 
reading and math. Some change in the students’ attitude toward school 
might have been induced because of improved achievement, but the pri- 
mavy objective was improved achievement. 

On the other hand, the involvement appeared to be oriented to 
improving the attitudes of both the students and their parents toward 
school, (Actually, the involvement also changed the teachers’ atti- 
tudes toward their students and toward the students’ parents as well,) 
Many of the project personnel believe that the most essential part of 
the whole R-3 program was the parental involvement because it opened 
lines of communication that had not existed before. The parental 
involvement was required for a variety of reasons. Of course, the 
parents’ consent was required for the students’ participation in the 
program, but there were also effects on the way the parents looked 
at the role of education in their children’s lives. Before the R-3 
program, the parents had lacked a realistic appreciation of modern 
requirements for entry skills into the job market. Many of them had 
grown up making a living on the farm or in the canneries, and they did 
not understand that these jobs were becoming far less important for 
their children than some of the more highly technical jobs that were 
developing in San Jose* This lack of appreciation, coupled with strong 
family ties, resulted in parental urging that the students quit school 
so that they could contribute to the family income. Many families had 
large numbers of children and needed this extra money. 
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Although it would seem that the involvement was primarily aimed 
at attitude change, some of the project personnel thought that the in- 
tensive involvement trips were really the most influential component 
for improvement in academic achievement, as well as for attitude change. 
They believe that this is because the trips provide a dramatic break 
with traditional instrvtction, that they get the students away from the 
negative triggers that they have learned in their prior school experi- 
ence. These people pointed out that even the limited intensive involve- 
ment trip that was conducted at the other school actually resulted in 
improvement in academic achievement. (We can think of a number of rea- 
sons for this; for instance, much of the school was involved in plan- 
ning this intensive involvement trip, and therefore the teachers* 
attitudes toward their students were changed, as well as the other way 
around.) Whether the involvement really was principal in improving 
academic achievement will have to be resolved by further experimenta- 
tion. Lacking this experimentation, all we can do for the present is 
to point out the effects of the two different assumptions on the choices 
that one would make among the components. The one assumption is that 
the improvement in academic achievement primarily came from the reading 
and math program, and the other assumption is that it was caused by the 
involvement. 

The third component, gaming/simulation, showed how the program in 
reading and math was related to the world of work. Thus, it reinforced 
the reading and math and it also effected attitude change. Therefore, 
we shall assume that gaming/simulation had both objectives. 

Figure 4 shows which of the components is assumed to have which 
primary objectives. As you see, academic achievement is primarily at- 
tained through the reading and math program, supported by gaming/simu- 
lation. A positive attitude toward school is induced by the involvement 
and also supported by gaming/simulation. And there is a question as to 
whether the involvement was important for improving academic achievement 
Let us now consider various aspects of measures of effectiveness. 

One aspect has to do with the effects of the program over a longer pe- 
riod of time than we have been talking about up to now. (Of course, 

R-3 has not been in operation long enough for long-run effects to have 
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been observed.) Figure 5 displays the rate of growth in achievement in 
reading by grade. Normal growth, which is represented by the dashed 
line, would be indicated if a student was achieving at the 5th-grade 
level in the 5th grade, at the 6th-grade level in the 6th grade, and 
so forth. The rate of growth for Mexican-American students (Coleman, 
et al., 1966) is shown by the solid line. Because the Coleman report 
gave reading achievement levels only for the 6th, 9th, and 12th grades, 
the growth rate by grade can be inferred only very roughly, as indicated. 

A program like R-3 is intended to raise the growth rate at least 
to normal and, ideally, to provide sufficient initial growth that the 
student makes up for prior years of underachieveraent. Figure 6 shows 
some of the possibilities. The R-3 program succeeded in the 8th grade 
in raising the students’ growth rate to 1.8 months per month, as repre- 
sented by the sharp peak, while in the 9th grade the growth rate was 
1.1 months per month. In the future the students may drop back to their 
originally falling achievement or to normal growth, or they may even 
continue to grow at a somewhat greater rate than normal. 

Another aspect of effectiveness is the effect of changes in one 
achievement measure on other measures not affected by the program di- 
rectly. Logically, one might think that if a student’s performance in 
reading has been improved, his performance in other subjects would also 
improve, particularly those such as science that require reading skills. 
Therefore, perhaps the R-3 program has also raised the students’ achieve- 
ment in science above normal growth, as suggested by the upper dash-dot 
line on Fig. 7. But it is also quite possible that the science program 
has suffered by comparison with the R-3 activities. In that case, the 
students’ performance in science may even drop below the normally low 
achievement for this population. Thus, we must measure the students’ 
achievement in aVl areas of interest so that we will know what the in- 
direct effects are, if any. 

Other measures also are needed, especially if the program involves 
more than, say, 100 students. In that case, we will need background 
data of the type that is being collected for the evaluation of all of 
San Jose’s compensatory programs (Rapp, et al., 1969). These data are 
required so that the effects of varying backgrounds on student achievement 
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can be isolated from the effects of the programs. The San Jose survey 
will assess background in five areas: the parent’s view of the child, 

the parent’s view of the school, family history, language patterns 
(important for Mexican-American populations), and economic status. 
Follow-up information of the type that will be gathered on those who 
participated in R-3 is also needed because, of course, planners are 
primarily concerned with longer-term effects. Recall that among the 
data to be gathered are dropout rate, police contacts, patterns of col- 
lege entrance, and patterns of vocational choice. 

Now let us compare the hypothetical effectiveness of the equal- 
cost alternatives that have been described previously, and that were 
chosen so that the decisionmaker may concentrate his deliberations 
on effectiveness. These comparisons will be made using two different 
assumptions. The first assumption is that the reading and math pro- 
grams were primarily responsible for gains in academic achievement. 
Figure 8, based on this assumption, shows growth in reading achievement 
in months per month versus the number of students involved in the al- 
ternative programs. The first point on the figure, labeled ’’all,” 
represents the 1.8 months-per-month gain attained by the 30 students 
in the R-5 program. (At present, of course, ’’all” is the safest al- 
ternative to choose because there are no data on the effectiveness of 
the others.) 

Now if the components are recombined into the equal-cost alter- 
natives, there will probably be changes in achievement growth and 
there will also be changes in the numbers of students involved. For 
example, although the gaming/ simulation activity might not induce as 
much achievement gain as would the total program, it could be provided 
to five times as many students. It might, however, be too close to 
the dashed line — indicating 1 month per month, or normal growth. The 
dashed line is critical because the State of California considers any 
achievement growth less than this to be unacceptable. This would mean 
that, although the involvement could be given to 90 children for an 
equal cost, it would not be acceptable if the achievement it induced 
fell below the critical minimum, as suggested. If the district is not 
required to meet a minim\im standard in achievement gain, it will be 
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^jssible to trade off achievemtnt gain on the one hand and the number 
of students reached on the other. This might be an important consider- 
ation if the schools need vicibility. In that case the gaming/simula- 
tion would always be the best choice, because it is the least expensive 
per student. 

The other two alternatives shown may more than meet the minimum 
requirement so that the choice between them would depend on whether 
one felt it more important to provide a higher rate of achievement to 
fewer students or a lower rate of achievement to more students. Because 
the reading and math program is not particularly innovative, one might 
be more interested in the gaming/ simulation plus reading and math, even 
though it can be given to only 60 students. 

Now suppose that we assume that the involvement was primarily re- 
sponsible for the achievement gain rather than the reading^'^nd* math. 
Figure 9 is based on this assumption. Again, all components working 
together account for a 1.8 months-per month growth in reading achieve- 
ment for 30 students, but reading and math and gaming/simulation have 
dropped because they have been assumed to contribute relatively less 
to reading achievement. In fact, either of these alternatives might 
not be acceptable because they appear to fall on the critical line. 

The involvement has moved above the line, and since the involvement 
applies to 90 students, it looks like a very attractive alternative. 

In fact, it would appear that the involvement might be superior to the 
combination of gaming/ simulation with reading and math, both in terms 
of achievement and number of students reached. On the other hand, in 
some districts it may not be possible to maintain discipline among the 
students for the extended period that they must be away from the school 
and home on the intensive involvement trips. If this is a significant 
problem for the district, the involvement would be an unsafe choice, 
and it would be better to select the gaming/ simulation and reading and 
math. 

Now let us include another measure of effectiveness in our analy- 
sis. Two measures of effectiveness — one, growth rate in reading, and 
the other, an index of attitude change for each program — are shown on 
Fig. 10. This index was derived by assuming that each alternative would 
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TWO MEASURES OF EFFECTIVENESS — FI RST ASSUMPTION 
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induce a change in attitude relative to the change induced by the R-3 

program. Thus, the index of attitude change for the R-3 program is 

•k 

unity, and the other alternatives have indices less than this. As 
before, gaming/ simulation alone looks risky because it is not being 
reinforced with backup programs. In addition, it may induce relatively 
little attitude change; reading and math look even poorer in this re- 
gard, while the involvement is too low on achievement gains. Note how 
the addition of the second measure supports the superiority of the com- 
bination of reading and math with gaming/simulation over reading and 
math alone (refer to Fig. 8). Thus, we may want to accept smaller num- 
bers of students and have reinforcing programs, as in the gaming/simu- 
lation and reading and math, where we buy achievement gain and attitude 
change for 60 students, or we may prefer to buy less of each for 150 
students with the gaming/simulation alone. Which one a decisionmaker 
chooses will depend upon whether he considers gains per student or 
numbers of students reached more valuable. 

Supposing we believe that the greatest contributor to achievement 
gain was the involvement. Then we^ would have a situation something 
like that depicted on Fig. 11. Now the choice is even more clearly in 
favor of the involvement over the gaming/simulation than before. The 
gaming/ simulation and reading and math cannot compete with it on any 
measure and only gaming/simulation alone is superior — in terms of num- 
bers of students only. 

We have gone through this kind of analysis to demonstrate how it 
can assist decisionmakers in choosing among alternative programs by 
considering effectiveness alone. It also demonstrates the value of 
being able to attach measures of effectiveness to alternatives. (Re- 
call that we could only hypothesize what the effectiveness would be.) 



A slight digression at this point will help to explain the ratio- 
nale behind this figure. Analysts have a tendency to lump all measur- 
ables in single indices for the sake of simplicity of manipulation and 
presentation. For example, the number of students in each alternative 
program might have been included in the indices. Although this would 
have made for a very pretty picture, it would have been almost impos- 
sible to interpret because too many variables would be combined in a 
single point. As far as possible, it is better to keep measures that 
are significant in their own right separate. 
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But even though the analysis had to be made in the absence of solid 
data, at the least it has made subjective judgments explicit and re- 
lated them to one another in an orderly way. 
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